From: https://osf.io/p5xsd/files/osfstorage

knitr::opts_chunk$set(echo = F)

0.1 Load data:

The permutation takes to long to allow for re-computing the code each time, so load it:

1. Objective measures

1.1. Consistency:

The mean area is calculated by adding together the area for each stimulus and dividing by 29. This unit is transformed into a percentage area taking into account the different pixel resolution of each participant.
Mean area = $(Summed area / 29) * 100 / Screen area $ Where: $Screen area = Xpixels * Ypixels$

Note: 29 since 29 stimuli

1.2 Permuted consistency

Replicate Rothen methods. Might take some time to compute.

Calculating chance levels of consistency To create permuted datasets for each participant: the 87 xy coordinates are randomly shuffled so they are no longer linked to the original data labels (“Monday”, “5”, “April”, etc.). The mean area of the triangles based on the shuffled coordinates is computed (as described above), and the whole process is repeated 1000 times to obtain a subject-specific distribution of chance levels of consistency. A z-score is calculated comparing the observed consistency against the mean and SD of the permuted data: $Z = [(observed consistency) – (mean consistency of permuted data)] / (SD of permuted data)$

Code retrieved from OSF (adapted here):

1.3. SD as in Ward

As in Ward:

“Specifically, the standard deviation of the x-coordinates and/or the standard deviation of the y-coordinates (measured across all trials) should exceed a proposed value of 0.075 for a normalized screen with width and height of 1 unit.”

“A participant who produced a horizontal straight-line form would have a very low standard deviation in the y-coordinates but a high standard deviation in x-coordinates, and a participant with a vertical line would have the reverse profile. A participant with a circular spatial form would be high on both. A participant who clicks randomly around the screen would also be high on both x and y standard deviation, but would fail the consistency tests (the triangles would be large).”

2. Line intersection

An idea I have is to look into the lines and order of the forms. I would exclude when lines crosses. (since we expect forms the lines crossing means no form is formed). Needs refinement.

I think that the number of stimuli per condition should be taken into account (i.e. 9 numbers, 7 days, 12 months). Hence would need to be divided by this number of stimulus.

In each condition the connected x and y generates a segment, hence the number of segment is length(stimuli)-1. Moreover, currently, each stimuli is connected by 3 segment, one for each (of the 3) repetition. So dividing by 3, we have the average number of segment corssings per condition. Next we sum these for each ID Ideally we should compute the number of crossings across the repetitions, in addition to make it more complex it would also be computationally more demanding, and I don’t beleive it would lead to a significant difference.

To do: - Maybe the easier would be to have the average number of crossing per segment.

IMPORTANT: data frame needs to be informed of stimulus order to make sense!

3. ROC to set thresholds

We consider the questionnaires as ground truth. Since the questionnaires are scalable we still need to determine a different thresholds for the questionnaire. Also some questions might be more relevant than others to “diagnose” synesthesia (see Syn_NR).

3.1. ROC Segment intersections

3.1.1. AUC

We computed the ROC between the questionnaire with NR selected questions and the per participants averaged segment interceptions.

4.1.2. ROC and threshold

Form the ROC we used Youden’s J statistic o calculate the best threshold. This statistic maximizes the difference between true positive rate (i.e. sensitivity) and false positive rate (i.e. specificity - 1). This led to a threshold of 0.09 with a sensitivity of 84.15 and a specificity of 56.88. See Figure X:

3.1.3. Cumulative density plot

Visualize it with cumulative empirical density plots. In this plots the x xis indicates the average number of intersections per segments. The y axis indicates the percentage of participant. Hence while 93 % of the self identified synesthetes have less than 0.0927958` intersection per segment, 56 % of not self-identified synesthetes do.

3.2. ROC Consistency

3.2.1. AUC

3.2.2. ROC and threshold

Form the ROC we used Youden’s J statistic o calculate the best threshold. This statistic maximizes the difference between true positive rate (i.e. sensitivity) and false positive rate (i.e. specificity - 1). This led to a threshold of 0.16 with a sensitivity of 80.49 and a specificity of 40.26. See Figure X:

3.2.3. Cumulative density plot

Visualize it with cumulative empirical density plots. In this plots the x xis indicates the average number of intersections per segments. The y axis indicates the percentage of participant. Hence while 93 % of the self identified synesthetes have less than 1 intersection per segment, 52 % of not self-identified synesthetes do.

3.3. Power

## 
##      Two ROC curves power calculation 
## 
##          ncases = 7134
##       ncontrols = 33495
##            auc1 = 0.7179126
##            auc2 = 0.5265125
##       sig.level = 0.05
##           power = 1
##     alternative = two.sided

4. Add oder Criteris

4.1. Questinnnaire Ward

The same but with the Questionnaire threshold as defined in Ward. This led to a threshold of 0.25 with a sensitivity of 0.85 and a specificity of 0.62.

4.2. Permuted z-score consistency